System and method for improving tuning using caller provided satisfaction scores

ABSTRACT

A system and method provide a way of improving customer satisfaction with an IVR by identifying tuning opportunities for applications of an IVR system based on customer satisfaction scores. The system and method compare portions of the IVR application to a customer satisfaction score obtained from a customer satisfaction survey. The comparisons show, statistically, which portions of the IVR application correlate to a low customer satisfaction score. A report is produced which identifies these areas that correlate to low customer satisfaction, and these areas may be tuned to improve customer satisfaction with the IVR.

PRIORITY claim

This U.S. Nonprovisional patent application claims priority to U.S.Provisional Patent Application Ser. No. 61/014,771, filed Dec. 19, 2007,entitled System and Method for Automatic Identification of TuningOpportunities Using Caller Provided Satisfaction Scores.

TECHNICAL FIELD

The technical field of this application is computerized interactivevoice response (IVR) systems.

BACKGROUND

IVR systems are a communications technology that allow a computer todetect voice and touch tones during a normal phone call. An IVR systemcan respond with pre-recorded or dynamically generated audio to furtherdirect callers on how to proceed with a customer service request. IVRsystems can be used to control almost any function where the interfacecan be divided into a series of menu choices. IVR systems are typicallyused to service high call volumes, reduce cost and improve the customerexperience. Examples of typical IVR applications include telephonebanking and credit card transactions.

IVR systems may be monitored and updated for any of several reasons. Forexample, an IVR system may be updated to keep pace with changingbusiness needs and changing customer service requests. Similarly, IVRsystems may be updated in terms of improving an IVR system'sperformance. IVR system performance may be defined in any of severalways. For example, IVR system performance may relate to deliveringbetter customer service by increasing customer satisfaction metrics.Similarly, IVR system performance may relate to improving customercontainment within the IVR, thereby reducing the number of callerstransferred to a live customer service representative (CSR). Regardlessof the motivating factor, updating an IVR system to improve itsperformance may be referred to as tuning the IVR system.

Because IVR systems are computerized systems that address large callvolumes and generate numerous data records, monitoring and analyzingsuch IVR systems may pose a challenge in terms of informationmanagement. Various analysis packages have been developed that allow anIVR system analyst to review data recorded from the IVR system and makedecisions regarding where best to tune the IVR system. Some suchanalysis packages segment calls from an IVR system by category so thatcompanies can see the most common types of calls and where they may needto correct or improve a current business practice. Other types ofanalysis packages may record audio files from caller interactions withthe IVR for future playback by an IVR system analyst. Still otheranalysis packages may map calls to graphically show their sequence andwhere callers may be experiencing difficulty with an IVR system.

While several analysis packages do exist to help IVR system analystsreview the performance of an IVR system, these analysis packages arelimited because they perform analysis from the IVR system's or IVRsystem analyst's point of view. These approaches do not explicitlyinclude the caller's experience with the IVR, which may indicate thecaller's state of mind, his environment, and the caller's hiddenrequirements that are not explicitly stated in the IVR. Hence, from theIVR system's or IVR system analyst's point of view, a call may beperfect with no errors, but the caller may still be unsatisfied. Hence,the outputs of the prior art analysis packages do not always address thecallers' needs. Certain embodiments described further herein aredesigned to provide a solution to one or more of the weaknesses in IVRtuning technology set forth above.

SUMMARY

Embodiments disclosed herein relate to a system and method for improvingcustomer satisfaction with an IVR system. In some embodiments, this isaccomplished by using customer satisfaction scores to identifyopportunities to tune an IVR, and then tune the IVR to improve customersatisfaction.

It will be appreciated by those skilled in the art that the teachingsdisclosed herein may be used to improve an IVR system's effectiveness interms of containment and other measures. Also, the teaching disclosedherein may be adapted to web based customer service applications. Morespecifically, the navigational patterns of the website may be linkedwith a user provided survey to identify deficiencies in the webapplication and enhance web content to better meet the user's needs.

Systems and methods of various embodiments described herein may beimplemented in a computer environment using computer readable mediums,computer executable instructions, and other suitable hardware. It shouldfurther be understood that the system and method disclosed herein may beassociated with an automated feedback loop. An example of such afeedback loop can be seen in U.S. Ser. No. 11/276,497, entitled SYSTEMAND METHOD FOR CLOSED LOOP DECISIONMAKING IN AN AUTOMATED CARE SYSTEM,filed Mar. 2, 2006, the disclosure of which is incorporated herein byreference. Associating the systems and methods disclosed herein with anautomated feedback loop may provide advantages in terms of datacollection, data processing, and/or data analysis.

All terms defined herein shall be read to include, but not be limitedto, all meaning ordinarily ascribed to that term by those of ordinaryskill in the art, and any additional meaning described. Before providingfurther description of exemplary embodiments, it will be instructive toprovide some term definitions.

The term “tuning,” shall be read to include, but not be limited to, anyadjustment of a system or process to improve performance.

The term “containment,” shall be read to include, but not be limited to,the duration a caller remains in an IVR system.

The term “event log,” shall be read to include, but not be limited to,any log that captures computer data. By way of example, and notlimitation, an event log for an IVR application may capture allinformation associated with a call. For instance, an event log may showeach dialog module visited, the result from each dialog module visited,calling frequency, results from previous calls, day and time of calls,call duration, number of failures in each dialog module, the speechrecognition result from each dialog module, etc. Additionally, an eventlog may capture data in a variety of formats. For example, an event logmay record audio data, text data, or text data derived from a computerconversion of audio data to text data.

The term “survey log” shall be read to include, but not be limited to,any log that captures survey data. By way of example, and notlimitation, a survey log for an IVR application may capture a caller'sresponses to survey questions asked by the IVR application.

The term “unique call identifier,” shall be read to include, but not belimited to, any manner of grouping calls from an IVR application so thateach portion of an event log is easily associated with the call fromthat portion of the event log. Those of ordinary skill in the art willappreciate that event logs do not typically record call information froman IVR application in sequenced order by call. Therefore, assigning aunique call identifier is a way to group the event log records insequenced order by call.

The term “simplified call flow log,” shall be read to include, but notbe limited to, a record of calls in an IVR system organized by call anddisplaying the calls according to their sequence of events in the IVRapplication.

The term “simplified compiled log,” shall be read to include, but not belimited to, a record of calls in an IVR system organized by call andmatched to the corresponding customer satisfaction score for the givencall.

The term “correlation,” shall be read to include, but not be limited to,the degree to which two or more attributes or measurements show atendency to vary together. The variation of correlated attributes ormeasurements may be positive or negative. When a positive correlationexists, as one attribute or measurement increases, the other attributeor measurement increases. Alternatively, when one attribute ormeasurement decreases, the other attribute or measurement decreases.When a negative correlation exists, as one attribute or measurementincreases, the other attribute or measurement decreases. Alternatively,when one attribute or measurement decreases, the other attribute ormeasurement increases.

The term “computer readable medium,” shall be read to include, but notbe limited to, any device or medium, or combination of devices andmedia, which is capable of storing computer readable and/or executableinstructions and/or data.

The term “computer executable instructions,” shall be read to include,but not be limited to, any combination of one or more computerinstructions regardless of how organized, whether into one or moremodules, one or more programs, a distributed system or any otherorganization.

The term “node traffic,” shall be read to include, but not be limitedto, the number of occurrences observed for a given prompt.

In considering the systems and methods described herein, it should beunderstood that the systems and method may include, but are not limitedto, any of the following versions.

In one version, a computer readable medium has computer executableinstructions to perform a method for improving customer satisfactionwith an IVR. In this version, the IVR has a customer satisfactionsurvey. One step of the method is processing an event log from the IVR.The processed event log includes call information, such as, prompt type,prompt response, and prompt result. The act of processing the event logresults in organizing the call information by unique call. Another stepincludes integrating a survey log with the processed event log. Thesurvey log includes a customer satisfaction score related to thecustomer's satisfaction with the IVR. The customer satisfaction score isprovided by the customer in response to the customer satisfactionsurvey. A next step is identifying a portion of the call information andmeasuring the occurrences for the portion of the call information. Thisportion of the call information is then analyzed using the occurrencesand customer satisfaction score to determine whether the portion of thecall information is associated with a low customer satisfaction score.

In another version, the method also includes creating a report. Thereport may include a table having a list of portions of call informationassociated with low customer satisfaction scores. This list may furtherbe prioritized by the customer satisfaction score. Alternatively, or inaddition, the report may include a graph that includes a series of leafnodes, where each leaf node represents an area of the IVR associatedwith the customer satisfaction score. Furthermore, the customersatisfaction score of the graph is based on an average of surveyresponses from a plurality of customers.

In another version, the act of analyzing the portions of the callinformation further comprise creating a correlation comparing a firststate where the portion of the call information was present in the eventlog to a second state where the portion of the call information was notpresent in the event log. Furthermore, the result of the correlation maybe weighted using the occurrences of the portion of the callinformation.

In another version, the portion of the call information may include anyof the following pieces of call information, singularly or incombination: prompt type, prompt result, prompt response, and dialogmodule result. Still in other versions, the portion of the callinformation may include a call path.

In some versions, where the portion of the call information includes aprompt response, the prompt response comprises a timeout. In someversions, where the portion of the call information includes a promptresult, the prompt result comprises a reject value, which indicates thatthe prompt response was not recognized by the IVR—also known as a “nomatch” scenario.

In another version, the act of processing the event log comprises: (a)identifying call timestamp information; (b) identifying an automaticnumber identification (ANI); (c) identifying an IVR application name;(d) creating a unique call identifier for each call based on thetimestamp information and the ANI; (e) assigning the unique callidentifier to each call; and (f) grouping the call information from theevent log by the unique call identifier.

In another version, the act of integrating the survey log with theprocessed event log comprises: (a) creating a common unique callidentifier for the survey log and the processed event log; and (b)matching the data from the survey log to the data from the processedevent log using the common unique call identifier.

In another version, a computer readable medium includes computerexecutable instructions to perform a method for improving customersatisfaction with an IVR having a customer satisfaction survey, themethod comprising: (a) accessing call events, and responses from acustomer to the customer satisfaction survey, in one or more logs; (b)selecting one or more prompts of the IVR and the associated informationfor analysis; and (c) correlating the selected one or more prompts andthe associated information with the matching responses to the customersatisfaction survey.

In another version, the method further comprises weighting the responsesfrom the customer satisfaction survey. Such weighting may be doneaccording to a customer's value. In some versions, the act of weightingcomprises: (a) measuring occurrences of the one or more prompts and theassociated information selected for analysis; and (b) artificiallyincreasing the observed occurrences by a factor commensurate with thecustomer's value prior to the act of correlating the one or more promptsand the associated information to the responses from the customersatisfaction survey.

In another version, a system exists for improving customer satisfactionwith an IVR, the system comprises: (a) a processor; (b) a userinterface; (c) a computer readable medium associated with the processor;and (d) a set of computer executable instructions, wherein the set ofcomputer executable instructions are encoded on the computer readablemedium, and the set of computer executable instructions when executed:(i) process a computerized event log, wherein the event log includescall information from the IVR; (ii) integrate a computerized survey logwith the event log, wherein the survey log includes a customersatisfaction score associated with the IVR; and (iii) identify callpaths from the IVR associated with low customer satisfaction scores,wherein the call paths are comprised of a plurality of prompts from theIVR.

BRIEF DESCRIPTION OF THE DRAWINGS

Further description of the embodiments is provided in the accompanyingdetailed description, which may be best understood in conjunction withthe accompanying diagrams, where like parts in each of the severaldiagrams are labeled with like numbers, and where:

FIG. 1 depicts a high-level flow diagram of a process for improvingperformance of an interactive voice response application.

FIG. 2 depicts a flow diagram for processing the event log of FIG. 1.

FIG. 3 depicts a flow diagram of a process for analyzing the simplifiedcompiled log of FIG. 1 in a prompt level analysis.

FIG. 4 depicts a flow diagram of a process for analyzing the simplifiedcompiled log of FIG. 1 in a call path level analysis.

FIG. 5 represents an exemplary high-level diagram of a system that maybe used with the methods disclosed herein.

FIG. 6 depicts an exemplary redacted processed event log of FIG. 1.

FIG. 7 depicts an exemplary survey log of FIG. 1.

FIG. 8 depicts an exemplary simplified compiled log of FIG. 1.

FIG. 9 depicts an exemplary data table derived from a simplifiedcompiled log.

FIG. 10 depicts an exemplary tuning opportunity report in graphicalform.

FIG. 11 depicts an exemplary tuning opportunity report in table form.

DETAILED DESCRIPTION

Reference should now turn to the figures, which depict a system andmethod for improving customer satisfaction with an IVR system.Improvements are achieved by identifying tuning opportunities in an IVRsystem using a customer satisfaction score, and tuning the IVRaccordingly. In the figures, like parts are designated with likenumerals throughout.

Referring to FIG. 1, a method is shown for improving customersatisfaction with an IVR application. Block 110 depicts the step ofproviding an event log 99 from an IVR application. In some versions, theevent log 99 may be provided to an IVR system administrator. The eventlog 99 may be provided as a data record on any suitable storage means,but preferably any suitable computer readable medium. Furthermore theevent log 99 may be provided locally or over a network connection. Inother versions, providing the event log 99 may encompass transmittingthe event log 99 from one system or data location to another system ordata location.

In considering the step of block 110, it should be appreciated that anIVR system may be comprised of one or more IVR applications, and anevent log 99 may be organized in a variety of ways. For example, anevent log 99 may be specific to a particular IVR application within anIVR system. Alternatively an event log 99 may capture information frommultiple IVR applications within an IVR system. By way of example, anIVR system may contain two IVR applications: a change of addressapplication and a store locator application. An event log 99 may existfor each application separately, or a single event log 99 may exist forboth applications. Those of ordinary skill in the art will appreciatethat the important aspect in this regard is to capture the eventinformation from the calls in any suitable organization of an event log99.

Additionally, it will be appreciated that event log 99 may berepresented by a variety of other names known in the art. For example,in some versions, an event log 99 may be the same as, or similar to, atuning log. Those of ordinary skill in the art will appreciate that theimportant aspect is not the nomenclature used to describe the log, butthe data captured in the log itself, as discussed below.

In block 112, the data from the event log 99 is processed to identifyunique calls and their process flows. It will be appreciated that theevent log 99 is a collection of call events recorded as the events occurwithin the IVR. For example, the event log 99 is not organized bycaller, but rather events are recorded as they occur within the IVR. So,events from separate calls may appear next to each other in the eventlog 99. Thus the data from the event log 99 must be further processed toorganize the data by unique call.

Referring to FIG. 2, the steps involved in processing the data from theevent log 99 are shown. In block 210, call timestamp information isidentified. Next, block 212 depicts a process for further callidentification using the call automatic number identification (ANI).Furthermore, block 214 depicts a process for identifying the IVRapplication name (APNM). Block 216 depicts associating a unique callidentifier to each call based on the timestamp, ANI, and APNM. Block 218depicts the step of organizing the calls in the event log 99 based onthe unique call identifier to produce a processed event log 101. Thoseof ordinary skill in the art will appreciate that the above descriptionprovides only one exemplary way to process the data of the event log 99to identify unique calls. Other approaches may be suitable as well, andthe exact method used should not be limited to only those approachesdescribed herein. Any approach that produces a process event log 101that can be subsequently analyzed by unique call would be suitable. Itshould further be appreciated that, after an initial programming, thisstep of processing the event log 99 may be automated when practicing themethod for subsequent event logs containing the same or similar datatypes and organizations.

Referring again to FIG. 1, in block 114, the data captured in theprocessed event log 101 may be limited to focus on main prompts of theIVR, thereby producing a redacted processed event log 103. Inparticular, the data may be limited to focus on those areas of the IVRrelevant to analyzing the impact on customer satisfaction. By way ofexample only, data fields that may be excluded may include processingtime descriptors. The reasoning here may be that processing timedescriptors have a low chance of truly impacting customer satisfactionso they can be ignored and excluded from the data set. It should beappreciated that the step in block 114 may be optional in some versions.Where the limiting step is used, however, the decision regarding whatdata fields to include may be made by an IVR system administrator orother individual designing the systems and methods disclosed herein. Itshould further be appreciated that, after an initial programming, thisprocess of limiting may be automated when practicing the method forsubsequent event logs containing the same or similar data types andorganizations.

In block 116, a simplified call flow is created by further processing ofthe redacted processed event log 103. To understand how this simplifiedcall flow is created, reference to an exemplary high-level IVRapplication exchange and configuration is helpful.

In the exemplary IVR high-level exchange, there is a prompt, response,and result. The prompt is provided to a caller by the IVR. The promptmay take a variety of forms. By way of example only, a prompt may askthe caller to input the reason for their call. In doing so, the IVR mayor may not provide a list of selections to choose from when respondingto the prompt. After the prompt is delivered to the caller, the callerprovides a response to the IVR. In some IVRs the caller's response maybe made by pressing the keypad on the telephone or by spoken word(s).After receiving the response from the caller, the IVR must interpret theresponse to produce a result. The result may frequently includeidentifying a subsequent prompt to provide to the caller, at which pointthis exchange process repeats.

In the exemplary IVR configuration, the IVR application is organizedinto one or more dialog modules. The one or more dialog modules maycontain one or more prompts. Each attempt in providing a unique promptto a caller and acquiring a response may be represented as a state. Byway of example only, an IVR application may be structured to permitattempting the same prompt twice before transferring the caller to alive agent. In such a case, the first presentation of the prompt may besaid to occur in state 1, while the second presentation of the promptmay be said to occur in state 2. Assuming, in this example, that theprompt and response exchange was unsuccessful in both attempts, theresults at state 1 and state 2 would be unsuccessful; also, the resultof the dialog module as a whole would be unsuccessful. It should benoted here that there may have been a sequence of successful promptsoccurring in the dialog module before the unsuccessful prompt wasencountered. Although there were some successful prompts in the dialogmodule, the result of the dialog module is nonetheless consideredunsuccessful due to the two failed attempts at the unsuccessful prompt.An exemplary structure similar to this example will be discussed laterwith respect to FIGS. 6-8.

Having provided an exemplary IVR application exchange and configurationabove, it should be understood that several approaches for configuringan IVR application may be suitable. Furthermore, the disclosure of theexchange and configuration above is exemplary only and should not beconsidered limiting.

Returning now to the simplified call flow creation of block 116, asimplified call flow may be created by capturing certain informationassociated with the dialog modules and prompts appearing in the calls ofthe redacted processed event log 103. For example, the information tocapture may include, among others: (1) the prompt name (PRNM), (2) theprompt type (PRTX), (3) whether the prompt was interrupted by thecaller—i.e., did the caller barge in (BRGN), (4) the time of any bargein (BTIM), (5) the mode of prompt response (MODE), (6) the promptresponse (TRTT), (7) the prompt result for a given state (RDEC), (8) thedialog module name (DMNM), and (9) the dialog module result (TSTT).

By way of example only, in an IVR application for a store locator, theapplication may include an alpha-numeric dialog module name (DMNM) ofM0500. The M0500 dialog module may include a “get zip code” prompt inthe IVR that states “please say or enter your five digit zip code.” Thecaller may respond by speaking “45202.” The IVR may accept this responseand move to a subsequent prompt. The prompt type (PRTX) in this examplemay be considered “get zip code.” The prompt name (PRNM) may beconsidered “initial,” possibly indicating this was the first prompt inthis dialog module. The prompt response (TRTT) may be considered“45202.” The prompt result (RDEC) may be considered “accept.” The modeof response (MODE) may be considered “SPCH,” indicating the response wasspoken. Assuming that no prompts in the dialog module were unsuccessful,the dialog module result (TSTT) may be considered “success.” Thus thisexchange could be represented by the following expression:

DMNM=M0500

PRNM=initial

PRTX=get zip code

MODE=SPCH

TSTT=45202

RDEC=accept

TRTT=success.

It should be understood that the above simplified call flow is merelyexemplary. Other prompts and the information associated with the promptsmay be included in addition, or in the alternative, to that described inthe above example. For instance, other prompt response modes (MODE) mayinclude designations of “tone” or “timeout.” Such a designation mayoccur where the caller either entered a response using the telephonekeypad or the caller did not provide a response to the promptwhatsoever. Other prompt results (RDEC) may include “reject,” “failure,”or “confirm.” A “reject” may occur where the IVR did not receive aresponse from a caller or the IVR received a response that wasunrecognized. A “failure” may occur after a prompt has been unsuccessfulfor a predetermined number of attempts. A “confirm” may occur when aresponse was received, but the confidence in the response was below apredetermined value (possibly from a lower than desired confidencemeasure of a speech recognition program).

It should be appreciated that these above exemplary combinations ofprompt response modes (MODE) and prompt results (RDEC) are not intendedto be exhaustive or limiting. For instance, a “timeout” prompt responsemode (MODE) may lead to a “confirm” prompt result (RDEC) where theprompt is repeated to the caller instead of a “reject” prompt resultwhere possibly an alternate prompt is provided to the caller. It willalso be appreciated from the above examples, that the informationassociated with the prompt response mode (MODE), prompt result (RDEC),etc. are not fixed. For instance, in one version the systemadministrator may configure a “timeout” to be reflected as an output ofthe prompt response mode (MODE). However, in another version the systemadministrator may configure a “timeout” to be reflected as an output tothe prompt result (RDEC). Those of ordinary skill in the art willappreciate the many options available with respect to prompts and theinformation associated with the prompts.

In some other versions, the information associated with the dialogmodules and prompts discussed above may be presented and understood asphrases. In such versions a phrase may represent all or a portion of theprompt and response exchange. For example, a phrase may represent theprompt type (PRTX) in combination with the prompt response mode (MODE)and prompt result at a given state (RDEC). As an example only, a phrasecould represent a prompt asking for a zip code along with the caller'slack of response within a prescribed time. In this example, the phrasemay be represented by the simplified expression: “get zipcode—timeout—failure.” Still in other versions some of ordinary skill inthe art will refer to portions of the prompt and response exchange asfeatures. Those of ordinary skill in the art will appreciate that thenomenclature used to describe the exchange in the simplified call flowis not essential. Instead, the nomenclature may be adapted to suit thesystem administrator's preferences when designing the simplified callflow. The important feature here is that the exchange between the IVRand caller is captured in a way where the data can be used for lateranalysis as discussed further below.

Another point in considering block 116 and creating the simplified callflow is that various techniques may be used to accomplish this step. Inone exemplary version, event log 99 may already capture the desired datain a usefully labeled form, and such event logs 99 may require verylittle additional processing after steps 112 and 114, if any, to arriveat a simplified call flow. For instance creating the simplified callflow may be a matter of further excluding irrelevant portions of theredacted processed event log 103. Still in another exemplary version,the captured data in the event log 99 may require more substantialprocessing steps to arrive at a simplified call flow after completingsteps 112 and 114. For instance, where an event log 99 does not capturethe data in a usefully labeled form, the data of the redacted processedevent log 103 may require re-coding. Such re-coding may involve applyingabbreviated designations to relevant portions of the IVR for lateranalysis. For example, where a prompt response mode (MODE) shows atimeout, this may be re-coded as a “no input.” Furthermore, where aresponse was received at the prompt response (TSTT) field, yet a“reject” prompt result (RDEC) was recorded, this may be re-coded as a“no match.” It should further be appreciated that, after an initialprogramming, this process of further processing to arrive at asimplified call flow may be automated when practicing the method forsubsequent event logs containing the same or similar data types andorganizations.

In block 118, the data from the simplified call flow is aggregated intoa one row format where each row represents the complete course of aunique call. This representation may be known as the simplified callflow log 105.

In block 120, summary statistics may be presented for each unique call.Summary statistics may include metrics such as the total number ofprompts, the number of accepted responses, the number of failedresponses, the number of rejected responses, the number of confirmedresponses, and the ratio of successful responses to total prompts. Thoseof ordinary skill in the art will further appreciate that this list ofsummary statistics is not exhaustive and other summary statistics may bepresented as well. Furthermore summary statistics may be presented for agrouping of calls in addition to, or in the alternative to, the summarystatistics presented for each unique call. At a macro level, the summarystatistics may provide insight as to overall IVR applicationperformance.

Block 122 depicts the step of providing a survey log 107 from a customersatisfaction survey related to the IVR application. The survey log 107is a collection of all call survey prompts and responses recorded as thesurveys are conducted. In some embodiments, the customer satisfactionsurvey will include a customer satisfaction score. The customersatisfaction score may be directly or indirectly related to theperformance of the IVR application.

One of ordinary skill in the art will appreciate that a customersatisfaction survey, incorporated into an IVR application, may bedesigned in any number of ways. For instance, the customer satisfactionscore may be based on the response to a single question or it may be acalculated score from a multitude of responses. By way of example, andnot limitation, an IVR application may play a prompt asking a customerto score his overall satisfaction with the IVR application. Such asurvey may ask the one or more survey questions at any area within theIVR application. For instance, a survey may be requested by the IVRafter the caller has completed the IVR application. In another example,the IVR application may prompt the caller to answer a survey beforetransferring the caller to a live agent/customer service representative(CSR). Customer satisfaction surveys may be structured to gatherresponses on any variety of scales that will be appreciated by those ofordinary skill in the art. By way of example only, the scale used may bea numeric scale such as 1 to 10, or 1 to 100. Alternatively the scaleused may be a letter scale such as “A,” “B,” “C” and so forth. Also,such surveys may acquire responses in numeric form, recorded audio form,or textual form. Those of ordinary skill in the art will appreciate themyriad of ways to structure customer satisfaction surveys.

In block 124, the data from the survey log 107 is integrated with thesimplified call flow log 105. To accurately integrate the data from thesurvey log 107, the data must be identifiable as from a unique call thatis listed in the simplified call flow log 105. In one example, the datafrom the survey log 107 may use the same timestamp, ANI, and IVRapplication name data fields to identify the call that the survey isassociated with. Thus, using the timestamp, ANI, and IVR applicationname, each unique survey from the survey log 107 may be matched with thecorresponding unique call from the simplified call flow log 105 toproduce a simplified compiled log 109.

Referring now to FIGS. 3 and 4, exemplary methods of analyzing thesimplified compiled log 109 are shown. These analyses may take variousapproaches and should not be limited to only the approaches discussedwith respect to FIGS. 3 and 4.

FIG. 3 shows an exemplary method for analyzing the simplified compiledlog 109 at the prompt level. The objective with this analysis is todetermine the impact of an exchange where the exchange is limited to agiven prompt, response, and result.

FIG. 4 shows an exemplary method for analyzing the simplified compiledlog 109 at the call path level. The call path level is where theexchanges from more than one prompt are combined, and the combination isanalyzed. The objective with this analysis is to determine the impact ofan exchange where the exchange extends to a combination of prompts,responses, and results. It should be appreciated that the prompts,responses, and results that make up a call path may be, but are notrequired to be, sequential prompts, responses, and results occurringwithin the calls.

As a first step in either the prompt level or call path level analysis,blocks 310 and 410 show the step of identifying and isolating upstreameffects that may skew the analysis for a known reason. An example of anupstream effect may relate to separating certain call types where acaller may be predisposed to give a low customer satisfaction score. Byway of example only, a caller may have been receiving numerous telephonesolicitations from his bank to sign-up for the bank's latest credit cardoffering. The caller may be frustrated with these numerous calls anddecide to call his bank to request placement on a “do not call list.”Upon calling his bank, the caller may be routed to an IVR. Although theIVR may be equipped to handle the caller's request, the caller may stillprovide low satisfaction responses to the survey based on hisfrustration experienced outside the IVR. In another example of anupstream effect, a small business owner may be frustrated by receivinghis telephone bill for his business at his home address instead of hisbusiness address. The caller may decide to call the telephone company torequest a change of address on file for his business. The caller may berouted to an IVR to handle his request. Although the IVR may be equippedto handle the caller's request, the caller may still provide lowsatisfaction responses to the survey based on his frustrationexperienced outside the IVR. Thus in the analysis, it may be appropriateto isolate these types of calls to gain a more accurate assessment ofcaller satisfaction with the IVR itself rather than having the caller'sexperience outside the IVR inaccurately influence their satisfactionscore for their experiences inside the IVR. The upstream effectsdescribed here are not exhaustive and those of ordinary skill in the artwill appreciate the myriad of upstream effects that may exist and beproperly identified and isolated when doing the analysis.

Now referring to FIG. 3, in block 312, the simplified compiled log 109is analyzed to measure the occurrences, within each call, of the promptsand information associated with the prompts. This step may focus oncollecting the occurrences for the prompts alone, regardless of theinformation associated with the prompts, e.g. prompt response or promptresult. For instance, the occurrences of just the “get zip code” promptmay be measured. In other embodiments this step may focus on collectingthe occurrences for the prompt plus specific associated information. Forinstance, the occurrences of the “get zip code” prompt with a “no input”response may be measured. In another example, the occurrences of the“get zip code” prompt with a “failure” result may be measured. Still inother embodiments, this step may focus on collecting the occurrences foronly the information associated with the prompts, and not the promptsthemselves. For instance, the occurrences of all “no input” responsesmay be measured, regardless of the prompt they are associated with. Inanother example, the occurrences of “failure” results may be measured.Those of ordinary skill in the art will appreciate that the measurementof occurrences may be structured to take any of a variety of forms asthe above examples show.

FIG. 3 further includes a summary creation step as shown in block 314.The summary created in block 314 may include a breakdown of theoccurrences measured for the prompts and information associated with theprompts. As will be discussed in further detail later, these summarystatistics may be useful in certain aspects of the analysis, e.g.prioritizing the analysis results by weighting the analysis by frequencyof the occurrence of a specific prompt, etc.

After the occurrences are measured and the summary created, block 316depicts a correlation step where the occurrences and associated customersatisfaction scores are compared. This correlation is conducted bycomparing two populations of data. The first population of data isrepresented by the calls where the selected occurrence was observed. Thesecond population of data is represented by the calls where the selectedoccurrence was not observed. The customer satisfaction data for thesetwo populations are compared to determine whether there is astatistically significant difference in customer satisfaction when theselected occurrence existed in a call compared to when the selectedoccurrence did not exist in a call.

By way of example, and not limitation, a simplified compiled log 109 mayinclude 500 unique calls. Within these calls the occurrence of theprompt and response combination for “get account number—no input” may bemeasured. It may have been discovered that 70% of the calls did notexperience this prompt and response combination, while the other 30% ofcalls did have this prompt and response combination. Furthermore, thecustomer satisfaction from the survey data may indicate that the same70% of calls had a mean customer satisfaction score of 60% while thesame 30% of calls had a mean customer satisfaction score of only 20%.These two data populations can be evaluated statistically to determineif the change in the customer satisfaction score is significant. If thechange is significant, then one could identify this prompt and responsecombination as an area of the IVR for tuning. The objective here wouldbe to modify the IVR to eliminate as many “no input” responses to the“get account number” prompt.

In the above example, it may be discovered by further examination of theIVR that the “get account number” prompt only allows ten seconds toinput a response. For callers who may not know the account numberwithout looking it up, the ten second timeframe may be inadequate, andthus cause the “no input” response. This may suggest modifying the IVRat this prompt to allow more time for a caller to provide a response,and/or possibly to change the prompt to allow a caller to input otheracceptable information that the caller may have readily available, e.g.billing street address, in lieu of an account number. By making suchmodifications, one may surmise that the occurrences of “no input” at the“get account number” prompt would decrease and the overall mean customersatisfaction would increase. This fact of course could be verified infuture analyses of a subsequent event log 99 and survey log 107.

Referring again now to the call path level analysis seen in FIG. 4, inblock 412, the simplified compiled log 109 is analyzed to measure theoccurrences, within each call, of the call paths and informationassociated with the call paths. This step may focus on collecting theoccurrences for the call path prompts alone, regardless of theinformation associated with the call path, e.g. prompt responses orprompt results. For instance, the occurrences may be measured for a callpath represented as “get account number→get zip code.” In otherembodiments this step may focus on collecting the occurrences for thecall path prompts plus specific associated information. For instance,the occurrences may be measured for a call path represented as “getaccount number—no input→get billing address—accept.” Still in otherembodiments, this step may focus on collecting the occurrences for onlythe information associated with the call path prompts, but not theprompts themselves. For instance, the occurrences may be measured for acall path represented as “no input→accept.” Those of ordinary skill inthe art will appreciate that the measurement of occurrences may bestructured to take any of a variety of forms as the above examples show.

FIG. 4 further includes a summary creation step as shown in block 414.The summary created in block 414 may include a breakdown of theoccurrences measured for the call paths and information associated withthe call paths. As will be discussed in further detail later, thesesummary statistics may be useful in certain aspects of the analysis,e.g. prioritizing the analysis results by weighting the analysis byfrequency of the occurrence of a specific call path, etc.

After the occurrences are measured and the summary created, block 416depicts a correlation step where the occurrences and associated customersatisfaction scores are compared. This correlation is conducted bycomparing two populations of data. The first population of data isrepresented by the calls where the selected occurrence was observed. Thesecond population of data is represented by the calls where the selectedoccurrence was not observed. The customer satisfaction data for thesetwo populations are compared to determine whether there is astatistically significant difference in customer satisfaction when theselected occurrence existed in a call compared to when the selectedoccurrence did not exist in a call. It will be apparent from the abovedescription that the correlating step of 416 is similar to thecorrelating step of 316. The distinction is that the occurrencesassociated with step 316 are for prompt level information while theoccurrences associated with the step of 416 are for call path levelinformation.

By way of example, and not limitation, a simplified compiled log 109 mayinclude 500 unique calls. As a threshold matter, it may be decided thatonly call paths occurring in more than 2% of the calls will be analyzed.Within these calls the occurrences may be measured for the call path“get account number—no input→get billing street address—no match.” Itmay have been discovered that 80% of the calls did not experience thiscall path, while the other 20% of calls did experience this call path.Furthermore, the customer satisfaction from the survey data may indicatethat the same 80% of calls had a mean customer satisfaction score of 75%while the same 20% of calls had a mean customer satisfaction score ofonly 10%. These two data populations can be evaluated statistically todetermine if the change in the customer satisfaction score issignificant. If the change is significant then one could identify thiscall path as an area of the IVR for tuning. The objective here would beto modify the IVR to eliminate as many “get account number—no input→getbilling street address—no match” call paths as possible.

In the above example, it may be discovered by further examination of theIVR that the “get account number—no input→get billing street address—nomatch” call path only allows ten seconds to input a response for the“get account number” portion of the call path. For callers who may notknow the account number without looking it up, the ten second timeframemay be inadequate, and thus cause the “no input” response. This maysuggest modifying the IVR at this portion of the call path to allow moretime for a caller to provide a response, and/or possibly to change thisportion of the call path to allow a caller to input other acceptableinformation that the caller may have readily available, e.g. accountholder last name, in lieu of an account number. Additionally, it may bediscovered that the “get billing street address” portion of the callpath may only accept spoken responses. With street addresses beingdiverse and sometimes have multiple syllable words, it may be difficultfor the associated speech recognition program to always accuratelyinterpret a caller's response. Furthermore, if a caller calls from anoisy environment, the noise may interfere with the speech recognitionprogram. This may suggest modifying the IVR at this portion of the callpath to allow keypad entry of all or a portion of the billing streetaddress. By making one or more of the above modifications, one maysurmise that the occurrences of the “get account number—no input→getbilling street address—no match” call path would decrease and theoverall mean customer satisfaction would increase. This fact of coursecould be verified in future analyses of a subsequent event log 99 andsurvey log 107.

With respect to the correlation steps in blocks 316 and 416, it shouldbe noted that statistical analysis computing software may be used andadapted to run the correlations for all the possible combinations ofeither prompts and/or their associated information, or call paths and/ortheir associated information. The use of a robust statistical analysissoftware ensures that each possible contributor to customer satisfactionis assessed. Furthermore, use of a robust statistical analysis softwaresaves time and resources in that a system administrator is not requiredto specifically identify the desired combinations of either prompts andtheir associated information, or call paths and their associatedinformation for the correlating steps.

In the analysis performed in blocks 316 and 416, a variety ofstatistical techniques may be employed to ensure that the relativecontribution of each prompt is captured. For example, a combination ofbivariate statistical techniques involving a computed Pearsoncorrelation (where the formula is analogous to the phi coefficient indichotomous cases) and chi-square tests may be used. Statisticaltechniques may, in addition or in the alternative, employ otherapproaches including computing a point biserial correlation coefficient,a Spearman's rank correlation coefficient, a Kendall tau rankcorrelation coefficient, and/or a Goodman and Kruskal's lambda. Thevariety of statistical techniques that may be employed to make the abovecorrelations will be appreciated by those of ordinary skill in the art.

As alluded to above, the correlation analysis may be automated by usingany suitable algorithm to conduct the statistical correlations. By wayof example only, and not limitation, suitable algorithms for theanalysis may include decision trees, SAS algorithms, a hidden Markovmodel (HMM), an artificial neural network (ANN), a Gaussian mixturemodel (GMM), and a k-means algorithm. Those skilled in the art will beable to appreciate other suitable algorithms for conducting the analysisof block 316.

Blocks 318 and 418 of FIGS. 3 and 4 respectively depict a step ofweighting the correlations. The weighting step of 318, 418 may use thenumber of occurrences to identify how frequent a given prompt andassociated information, or call path and associated information, occurs.This frequency may be used as a factor to adjust the relative importancewhen considering tuning opportunities based on the correlations alone.For example, if a specific prompt and response combination show a strongcorrelation to a low customer satisfaction score, but the specificprompt and response combination rarely occur, then the tuningopportunity for this combined prompt and response may take a lowerpriority compared to other prompt and associated information that occurmore frequently yet may have weaker correlation to low customersatisfaction score.

Similarly, the weighting step may use a customer value engine (alsocalled a lifetime value optimizer engine) such as described in U.S.patent application Ser. No. 11/686,812, entitled SYSTEM AND METHOD FORCUSTOMER VALUE REALIZATION, filed Mar. 15, 2007, the disclosure of whichis incorporated herein by reference. In an example here, the weightingstep may use a customer value engine to provide more weight to surveyresponses from certain callers based on the caller's importance, orvalue to the company. By way of example only, and not limitation, theanalysis program may be constructed to replicate important caller'sresponses by a certain factor such that the results will be more heavilyweighted to such a caller's feedback. It should be understood that acustomer value engine may be used for a variety of things, including,but not limited to, determining a customer's importance or priority. Thecaller's importance may be based on a variety of factors, such as thecaller's net worth, the frequency of business transactions, or theamount of business the caller transacts with the company. Other factorsfor determining a caller's importance will be apparent to those skilledin the art.

Blocks 320 and 420 depict a step of identifying areas for IVR tuningbased on the correlations and weighting steps of blocks 316, 318 and416, 418. The identification of tuning opportunities in blocks 320, 420may occur in a variety of forms. In one embodiment this identificationof tuning opportunities occurs automatically. In such a scenario, areport may be automatically generated as an output of the statisticalcorrelation analysis and sent to an IVR system administrator for review.The report may include a prioritized list of tuning opportunities basedupon the correlation and weighting analysis described above. The reportmay be in graphical form or table form for easy viewing andinterpretation by the IVR system administrator. In some versions, aprioritized list of tuning opportunities may use a rules format wherethe significant and prioritized results are shown as a set of rules. Byway of example, and not limitation, a rule might include “moreoptions—reject >0” along with an associated customer satisfaction scoreof 20. This rule would suggest that a significant difference in customersatisfaction score exists when a caller experiences a rejected responsefrom the IVR at a “more options” prompt. And, based on the data set, theaverage customer satisfaction score will be 20 when a caller experiencesa “reject” result at a “more options” prompt.

Referring now to FIG. 5, a system for implementing an exemplary methodis shown. An event log 99 and a survey log 107 are used in a computersystem 500. The computer system 500 includes a computer readable medium510, processor (not shown), and one or more user interface devices (notshown). The computer readable medium 510 is associated with a first setof computer executable instructions 512 and a second set of computerexecutable instructions 514. The output from computer system 500 may bea tuning opportunity report 516.

The first set of computer executable instructions 512 may be designed tostore the event log 99 and survey log 107 in the computer readablemedium 510. It will be appreciated that the first and second set ofcomputer executable instructions 512, 514 may themselves be stored inthe computer readable medium 510 or in any other suitable location. Thesecond set of computer executable instructions 514 may be designed tomerge the event log 99 and survey log 107 to produce the simplifiedcompiled log 109 shown in the process step 124 of FIG. 1.

Furthermore, the first or second set of computer executable instructions512, 514 may contain the algorithm used in the analysis steps shown inFIGS. 3 and 4. The computer executable instructions 512, 514 may bedesigned to use the algorithm to analyze the simplified compiled log 109and generate a tuning opportunity report 516 identifying opportunitiesfor improving the IVR application as shown in the process steps 320 and420 of FIGS. 3 and 4 respectively.

It will be appreciated by those skilled in the art that a variety ofcomputing designs including hardware and software components may beadapted to perform the disclosed methods. It will further be appreciatedthat the systems and methods may be implemented in both networked andnon-networked environments. The above disclosed system as shown in FIG.5 is merely one representation of such a system and is not intended tobe limiting. For example, one or more computer readable mediums may beused when performing the methods. The computer readable mediums mayoperate with any suitable processor and user interfaces to execute themethod steps of the computer executable instructions. Also, a suitablesystem may include numerous sets of computer executable instructionseach designed to perform different steps of the methods disclosed.Alternatively, one set of computer executable instructions may bedesigned to perform all method steps. In addition, the algorithm foranalyzing and detecting low caller satisfaction events may be completelyautomatic, and thus not require human supervision. In some versions, anautomatic analysis approach such as this may be integrated into a liveproduction IVR where it will detect issues with the applicationdynamically on a rolling basis and will raise alerts when lowsatisfaction events are detected. In other embodiments, the analysisconducted by the algorithm may be triggered manually by an IVR systemadministrator on a periodic or as needed basis.

Referring now to FIGS. 6-11, these figures show exemplary aspects of thesystem and method disclosed herein. While some aspects appearing inFIGS. 6-11 may be consistent among figures, not all aspects shown areintended to be consistent among the figures.

FIG. 6 shows an exemplary redacted processed event log 103 derived froman event log 99. As discussed above, the redacted processed event log103 is derived by first processing the event log 99 to identify uniquecalls, and then limiting the data to main prompts. In this redactedprocessed event log 103, there are three exemplary dialog modules of anIVR represented. The caller first goes through a first dialog module,abbreviated by the dialog module number (DMNM) M0400. Then the callergoes through a second dialog module, abbreviated by the dialog modulenumber M0280. Then the call goes through a third dialog module,abbreviated by the dialog module number M0440.

At the M0400 dialog module, a first state is started as represented bythe data “EVNT=SWIstst.” The next event down, “EVNT=SWIprst,” indicatesthe start of a prompt. The prompt name (PRNM) is “INITIAL_PROMPT,” andthe prompt type (PRTX) is “m0400_initial.ulaw.” The end of the prompt isrepresented by the data “EVNT=SWIendp.” Furthermore, it is clear thatthere was no “barge in” that ended this prompt prematurely as shown bythe data, “BRGN=0.” The prompt response here is seen as“TRTT=open_(—)24_hours.” The caller provided this response by speech ascaptured by the data, “MODE=SPCH.” The prompt result at this state(RDEC) is seen as “accept.” At that point, there is also an end to thestate as seen by the data “EVNT=SWIstnd.” Finally, the dialog moduleresult is provided by “TSTT=Success.” In summary form, at the M0400dialog module, the caller was provided the prompt abbreviated by m0400initial.ulaw. The caller responded by voice, indicating the“open_(—)24_hours” option. This response was accepted and this dialogmodule ended successfully.

The M0280 dialog module follows this same scheme. However, in thisdialog module, it can be seen that the caller did not respond to theprompt as indicated by the data “MODE=Timeout.” This is further evidentby the prompt response (TRTT) not including any recorded data. In thisdialog module example, only a single timeout was permitted before theentire dialog module is considered unsuccessful. As such, the dialogmodule result (TTST) is provided as “Max timeouts.”

The third dialog module shown, M0440, shows that the caller provided aprompt response (TRTT) of “survey.” This response was accepted as shownby “RDEC=accept.” Also in this dialog module, the caller interrupted theprompt as seen by the data “BRGN=1.”

FIG. 7 shows an exemplary survey log 107. In this survey log 107, thereare three exemplary dialog modules of the IVR represented. Here, eachdialog module in the survey represents a different survey question forthe caller. In FIG. 7, a first dialog module pertains to a qualityquestion, as seen in the abbreviated dialog module number A100_Quality.At the conclusion of the dialog module is the associated customersatisfaction score, in the example shown in FIG. 7 the customersatisfaction score is five (TRTT=5). The caller's response to the prompthere was accepted as seen by “RDEC=accept.” Furthermore, this dialogmodule was successful as seen by “TSTT=Success.”

A second dialog module pertains to how effectively the caller's questionor problem was resolved, and is seen in the abbreviated dialog modulenumber A100_Resolution_Navigation. In this dialog module, the callerinterrupted the prompt type (PRTX)A100_resolution_navigation_initial_ulaw. After the interruption, aprompt result (RDEC) of “confirm” was given. Thus, the IVR then provideda prompt type (PRTX) of “you_said.ulaw.” The caller here did not confirmthe IVR's understanding as seen by “DCSN=Denied.” The IVR then provideda wrong answer apology prompt and then replayed the original prompt type(PRTX) A100_resolution_navigation_initial_ulaw. In this second attempt,the caller responded with a customer satisfaction score is five(TRTT=5). The caller's response to the prompt here was accepted as seenby “RDEC=accept.” Furthermore, this dialog module was successful as seenby “TSTT=Success.”

A third dialog module pertains to an instructions questions, as seen inthe abbreviated dialog module number A110_Front_End_Instructions and theprompt type (PRTX) of A110_front_end.ulaw. At the conclusion of thedialog module is the associated customer satisfaction score, in theexample shown in FIG. 7 the customer satisfaction score is five(TRTT=5). Again, the caller's response to the prompt here was acceptedas seen by “RDEC=accept.” Furthermore, this dialog module was successfulas seen by “TSTT=Success.”

By way of example only, a quality question in a survey as shown by thesurvey log 107 of FIG. 7 may ask, “On a scale of 1 to 10 please rate thequality of your experience with this customer support application.”Similarly, a resolution question may ask, “On a scale of 1 to 10 pleaserate how well this customer support application resolved your issue.”Likewise, an instructions question may ask, “On a scale of 1 to 10please rate the instructions given during this customer supportapplication.” The responses to these survey questions may be combined toform a single overall customer satisfaction score, e.g. the responsesmay be averaged to produce an overall statistic, however othercalculating methods may be used to arrive at an overall customersatisfaction.

FIG. 8 shows an exemplary simplified compiled log 109 (albeit withoutthe data aggregated to one-row per call due to printing spaceconstraints). In FIG. 8, the redacted processed log 103 of FIG. 6 ismerged with the survey log 107 of FIG. 7. The result of the merging ofthe two logs is a simplified compiled log 109. As discussed above, whencreating the simplified call flow log 105 and ultimately the simplifiedcompiled log 109, sometimes it may be beneficial to re-code relevantdata for clarity and ease of analysis. However, such re-coding is notalways required or necessary. In the example of FIGS. 6-8, the redactedprocessed event log 103 and simplified call flow log 105 wereessentially synonymous such that creating the simplified compiled log109 only required combining the data from the survey log 107. Referringto FIG. 8, the 3 dialog module exchanges of FIG. 6 appear at the topportion of FIG. 8, while the three dialog module exchanges for thesurvey from FIG. 7 appear at the bottom portion of FIG. 8. Again, whereprinting space limitations are not an issue, all the data represented inFIG. 8 would be shown in a single row format since the information isfrom a unique call.

FIG. 9 shows an exemplary data table 910 derived from a simplifiedcompiled log 109. In some versions this data table may be referred to asa feature vector table. The data table 910 presents the informationcaptured in the simplified compiled log 109 in a form suitable forfurther analysis by any suitable computer programming algorithm asdiscussed above. In the left-most column, FIG. 9 shows five differentprompts represented in abbreviated numeric form: h1_a0210, h1_m0100,h1_m0113, h1_m0420, and h1_y0150. Also shown in the left-most column arethe four different results for these prompts: present, accept, timeout,and reject. A series of twenty calls and their occurrences for theprompt and result combinations are shown in FIG. 9. A “0” in the tablemay represent that there was no occurrence for that caller for thatspecific prompt and result combination. Similarly, a “1” in the tablemay represent that there was at least one occurrence for that caller forthat specific prompt and result combination. Additionally, thecorresponding customer satisfaction score, from the initial survey log107 and later recorded in the simplified compiled log 109, is shown foreach call.

One exemplary analysis for the data table 910 of FIG. 9 may be conductedby creating two populations of data and then performing statisticalsignificance testing. For example, one may focus on the prompt andresult combination of h1_m0113 accept. The two populations here would bedivided into (1) those that did not have this particular prompt andresult combination and (2) those that did have this particular promptand result combination. The customer satisfaction scores from these twopopulations would then be compared to determine if there was astatistically significant difference in the customer satisfaction scorefor these two populations. Those of ordinary skill in the art willappreciate that this example is merely a snapshot of the type ofanalysis that may be conducted.

FIGS. 10 and 11 show exemplary tuning opportunity reports 516. FIG. 10shows a graphical representation of a tuning opportunity report 516 froma decision tree analysis. As shown in FIG. 10, each leaf node representsa branch of the analysis for consideration. For example, at the top, of283 callers, 7 experienced a reject result at the h1_a0210 prompt. Theaverage customer satisfaction score for these 7 occurrences is 20%. Ofthe 276 remaining callers that continued to the h1_m0100 prompt, 9experienced a timeout result at the h1_m0100 prompt. The averagecustomer satisfaction score for these 9 occurrences is 22%. Continuingthis progression, as shown in the report 516, compared to the customersatisfaction scores at the other leaf nodes, the customer satisfactionscores at leaf nodes 1 and 2 are the lowest and thus may representpotential tuning opportunities.

However, one must still consider the prioritization aspects, namely thefact that focusing on leaf nodes 1 and 2 only address 16 out of 283callers—this is less than 6% of the callers. Arguably, it may havegreater impact to improving overall customer satisfaction by focusing onthe 28 callers (almost 1.0% of the callers) at leaf node 3 and trying tomodify the IVR to prevent these callers from having a timeout at theh1_m0113 prompt. These prioritization decisions may be made in a varietyof ways. For instance, such decisions may be made by a systemadministrator responsible for improving customer satisfaction with theIVR. In other examples, such prioritization occurs by rank orderingmathematically. Such mathematic rank ordering may take advantage of anynumber of rank ordering formulas which will be apparent to those ofskill in the art. As discussed above some rank ordering factors mayinclude the frequency of occurrences and the customer's value.

FIG. 11 shows an exemplary tuning opportunity report 516 in tableformat. The rules column in FIG. 11 contains rule statements that areassociated with a customer satisfaction score. For example, the firstrow shows a customer satisfaction score (CSAT) of 7 from thoseexperiencing a reject result at the h1_a0210 prompt. Also shown in FIG.11 is the number of surveys used for calculating the average customersatisfaction score, as well as the node traffic. The node traffic metricmay be helpful in weighting the customer satisfaction score whenprioritizing tuning opportunity areas. For instance, a tuningopportunity may receive a lower priority where a rule is associated witha low node traffic percentage. This is because the low node trafficpercentage means that few callers are experiencing the particularoccurrence represented by the rule. Thus the greatest impact onimproving customer satisfaction may be achieved by focusing on thoseareas showing relatively high node traffic with a relatively lowcustomer satisfaction score.

In summary, numerous benefits have been described which result fromemploying the concepts disclosed herein. The foregoing description ofone or more of the embodiments has been presented for purposes ofillustration and description. It is not intended to be exhaustive or tolimit the invention to the precise form disclosed. Obvious modificationsor variations are possible in light of the above teachings. The one ormore embodiments were chosen and described in order to best illustratethe principles and practical application of the disclosure to enable oneof ordinary skill in the art to best utilize the various embodiments,with or without various modifications as are suited to a particularcontemplated use. It is intended that the scope of the invention bedefined by the claims appended hereto.

We claim:
 1. A non-transitory computer readable medium comprisingcomputer executable instructions to perform a method for improvingcustomer satisfaction with an IVR having a customer satisfaction survey,the method comprising: a. processing an event log from the IVR toproduce a processed event log, wherein the processed event log includescall information comprising a prompt type, a prompt response, and aprompt result, wherein the act of processing comprises organizing thecall information by call; b. integrating a survey log with the processedevent log to produce a simplified compiled log, wherein the survey logincludes a customer satisfaction score by call, wherein the customersatisfaction score is associated with the customer's satisfaction withthe IVR, wherein the customer satisfaction score is provided by thecustomer in response to the customer satisfaction survey; c. identifyinga portion of the call information from the simplified compiled log foranalysis; d. measuring occurrences for the portion of the callinformation; e. analyzing the portion of the call information todetermine whether the portion of the call information is associated witha low customer satisfaction score, wherein the act of analyzingincorporates the occurrences for the portion of the call information andthe customer satisfaction score for the call; and f. creating a report,wherein the report includes a table having a list of portions of callinformation associated with low customer satisfaction scores, whereinthe list is prioritized by the customer satisfaction score.
 2. Thecomputer readable medium of claim 1, wherein the method furthercomprises creating a graph, wherein the graph includes a series of leafnodes, wherein each leaf node represents an area of the IVR associatedwith the customer satisfaction score, wherein the customer satisfactionscore is based on an average of survey responses from a plurality ofcustomers.
 3. The computer readable medium of claim 1, wherein thecustomer satisfaction survey comprises one question associated with theIVR.
 4. The computer readable medium of claim 1, wherein the callinformation further comprises a dialog module result.
 5. The computerreadable medium of claim 1, wherein the act of analyzing the portions ofthe call information further comprise creating a correlation comparing afirst state where the portion of the call information was present in theevent log to a second state where the portion of the call informationwas not present in the event log.
 6. The computer readable medium ofclaim 5, wherein the method further comprises weighting the result ofthe correlation using the occurrences of the portion of the callinformation.
 7. The computer readable medium of claim 6, wherein theportion of the call information comprises a combined prompt type andprompt result.
 8. The computer readable medium of claim 6, wherein theportion of the call information comprises a combined prompt type andprompt response.
 9. The computer readable medium of claim 8, wherein theprompt response comprises a timeout.
 10. The computer readable medium ofclaim 6, wherein the portion of the call information comprises acombined prompt type, prompt response, and prompt result.
 11. Thecomputer readable medium of claim 10, wherein the prompt resultcomprises a reject value, wherein the reject value indicates that theprompt response was not recognized by the IVR.
 12. The computer readablemedium of claim 6, wherein the portion of the call information comprisesa call path.
 13. The computer readable medium of claim 1, wherein theact of processing the event log comprises: a. identifying call timestampinformation; b. identifying an automatic number identification (ANT); c.identifying an IVR application name; d. creating a unique callidentifier for each call, wherein the unique call identifier is basedthe group comprising the timestamp information and the ANI; e. assigningthe unique call identifier to each call; and f. grouping the callinformation from the event log by the unique call identifier.
 14. Thecomputer readable medium of claim 13, wherein the act of integrating thesurvey log with the processed event log comprises: a. creating a commonunique call identifier for the survey log and the processed event log;and b. matching the data from the survey log to the data from theprocessed event log, wherein the act of matching uses the common uniquecall identifier.
 15. A non-transitory computer readable mediumcomprising computer executable instructions to perform a method forimproving customer satisfaction with an interactive voice response (IVR)having a customer satisfaction survey, the method comprising: a.accessing call events, and responses from a customer to the customersatisfaction survey, in one or more logs; b. selecting one or moreprompts of the IVR and the associated information for analysis; c.correlating the selected one or more prompts and the associatedinformation with the matching responses to the customer satisfactionsurvey; d. weighting the responses from the customer satisfaction surveyby a customer's value, wherein the act of weighting comprises: i.measuring occurrences of the one or more prompts and the associatedinformation selected for analysis; and ii. artificially increasing theobserved occurrences by a factor commensurate with the customer's valueprior to the act of correlating the one or more prompts and theassociated information to the responses from the customer satisfactionsurvey.
 16. A non-transitory computer readable medium comprisingcomputer executable instructions to perform a method for improvingcustomer satisfaction with an IVR having a customer satisfaction survey,the method comprising: a. processing an event log from the IVR toproduce a processed event log, wherein the processed event log includescall information comprising a prompt type, a prompt response, and aprompt result, wherein the act of processing comprises organizing thecall information by call; b. integrating a survey log with the processedevent log to produce a simplified compiled log, wherein the survey logincludes a customer satisfaction score by call, wherein the customersatisfaction score is associated with the customer's satisfaction withthe IVR, wherein the customer satisfaction score is provided by thecustomer in response to the customer satisfaction survey; c. identifyinga portion of the call information from the simplified compiled log foranalysis; d. measuring occurrences for the portion of the callinformation; e. analyzing the portion of the call information todetermine whether the portion of the call information is associated witha low customer satisfaction score, wherein the act of analyzingincorporates the occurrences for the portion of the call information andthe customer satisfaction score for the call; and f. creating a graph,wherein the graph includes a series of leaf nodes, wherein each leafnode represents an area of the IVR associated with the customersatisfaction score, wherein the customer satisfaction score is based onan average of survey responses from a plurality of customers.